Integrating Features, Models, and Semantics for Content-based Retrieval

In this paper, we describe a system for automatic and interactive content-based retrieval of video that integrates features, models, and semantics. The novelty of the approach lies in the (1) semi-automatic construction of models of scenes, events, and objects from feature descriptors, and (2) integration of content-based and model-based querying in the search process. We describe several approaches for integration including iterative filtering, score aggregation, and relevance feedback searching. We also describe our recent effort of applying the content-based retrieval system to the TREC video retrieval benchmark and report on our experience from combining features, models, and semantics.